The development of virtual reality technology has rapidly increased over the last few years. As virtual reality technology improves, new virtual reality applications in an array of fields including, for example, education and training, art, architectural design, movie production, advertisement, entertainment, video games, therapy, and medical fields, have been adopted or shown potential interest. The implementation of virtual reality in each field however often presents new challenges and requirements for the virtual reality application to be useful and functional. In resolving those challenges to meet each application's requirements, data content exponentially increases, compromising delivery reliability, feasibility, speeds and resolution.
Virtual reality experiences require high resolution content that is reliably delivered to the user for the experience to by fully immersive. Without a fully immersive experience, users of virtual reality are unlikely to adopt and implement the virtual reality technology for their particular application. In nearly all current virtual reality applications, the constraint on resolution generally results from the bandwidth available, as opposed to any limit on content generation. Depending on the playback device, whether it is streaming content or accessing locally stored content, and the application, additional constraints also result from the image processing and resolution due to limitations of the central processing unit (CPU), memory, screen maximum resolution, graphics processing unit (GPU) and storage capacity, power consumption, and overheating of the electronic device. These constraints in most cases decrease the level of immersion for the user and greatly compromise the reliability needed for some applications.
Recent image and video encoding and compression efforts have been explored. None of these however have resulted in a widely adopted solution due to fundamental limitations of their own. For example, various virtual reality content compression algorithms and methods specifically designed for 360-degree video have been experimented with. None have gained widespread use, nor have they adequately demonstrated their improvement over, or compatibility with, traditional video compression methods. U.S. Pat. No. 6,005,611 to Gullichsen et al., for example, describes a method that seeks to provide efficient delivery of 360-degree content where warped fish-eye lens video is stored and locally distorted depending on viewing geometry. This method by its own solution does not encode the full 360° of content, but rather only transmits and dewarps a rectangular subset of the data, irrecoverably limiting the delivered content. Because the content is irrevocably limited to only the subset transmitted, the viewer will experience dislocated views outside the transmitted region, disrupting immersion.
Alternative methods have been described in “Data compression on the sphere” http://arxiv.org/pdf/1108.3900.pdf and “Low bit-rate compression of omnidirectional images” http://infoscience.epfl.ch/record/130365/files/omni_cr.pdf, both which seek to compress data still in its spherical form. These too however have failed to be widely adopted in virtual reality applications due to the lack of flexibility between applications and projection devices. As another example of an unsuccessful compression idea for virtual reality content, U.S. Pat. Pub. No. 2016/0021373 A1 to Queru describes a method that only works with JPEG images, excluding all other forms of virtual reality content. In particular, because the method changes the number of macroblocks spatially, it must be fully integrated with the compression algorithm, rather than being an independent preprocessing step. This method also introduces compression artifacts which appear blocky and greatly hinder immersion, especially when present in stereoscopic content, as well as restricting the final resolution to be constant within all individual macroblocks. A common fundamental problem of the aforementioned compression methods is that traditional compression algorithms have hardware support for encoding and decoding on appropriate devices, so trying to force adoption of new compression algorithms is challenging.
Due to the aforementioned requirements and limitations, the prior art solutions thus suffer in their ability to enable processing and/or transmitting of high resolution virtual reality content to a user while also reducing the amount of content processed and bandwidth without compromising reliability. Accordingly:
There is a need for encoding and compression agnostic virtual reality content image processing solutions;
There is a need for image processing solutions that are flexible as to the projection functions that can be used to process and deliver virtual reality content;
There is a need for image processing solutions that result in useful distributions of resolution, such that data is uniformly and maximally useful;
There is a need for a virtual reality content processing and delivery method that can provide content flexibility and lower bandwidth requirements without compromising the virtual reality experience, for example, due to lower resolution and/or playback problems;
There is a need for a virtual reality content processing and delivery method that lowers the hardware and software requirements of processing and display devices; and
There is a need for image/video processing solutions that can reduce the amount of data processed by the devices to improve battery life and prevent overheating of the device.
Accordingly, improved virtual reality content processing and delivery methods and systems capable of overcoming the aforementioned needs are desired.